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INTRODUCTION 

The present invention relates generally to data processing systems, and 
more particularly to a format for holding and/or describing multimedia content that 
may include program instruction code for controlling the playback of the multime- 
dia content. 

BACKGROUND 

There are many file and/or stream formats in the technical field of the pre- 
sent invention. To mention a few: 
o The HTML standard 
o The MPEG-4 standard 

o Apple QuickTime Format (US patent number 5,751 ,281 ) 
o Microsoft ASF Format (US patent number 6,04:1 ,345) 
o Macromedia SWF Format (http://www.openswf.org) 
These formats are typically used to hold and describe multimedia content 
for use on the Internet. The file or stream based on the format is transmitted over a 
network to a destination computer containing a renderer, which process and ren- 
ders the content. Historically these formats were typically designed and implemen- 
ted for destination computers with good hardware resources (CPU, memory, disk, 
graphics card, etc.), such as personal computers (PCs). Typically, most of these 
formats support media types, such as images arid text. Some support video, 
audio, and 3D graphics. 

Conventional file and/or stream formats for holding and/or describing multi- 
media content that may include program instruction code for controlling the play- 
back of the multimedia content, are limited in several respects. First, these formats 
typically do not consider that the content may need to be used on any class of 
computer, from computers with very limited hardware resources (CPU, memory, 
disk, graphics card, etc.), to computers with powerful hardware resources. Such 
formats, typically require a renderer implementation that will be too large in 
> amount of disk or memory taken up by its program instruction code, or use too 
much of the hardware resources, for computers with very limited hardware 
resources (such as handheld devices). Another limitation of such formats is that 
they are generally limited in the lack of flexibility for representing different media 
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types. Such formats use quite limited predefined multimedia content types. They 
typically do not support real 3D graphics (textured polygon mesh), which is import- 
ant with respect to illustrating physical objects in a multimedia rendering. 

Yet another limitation of such formats is that they typically cannot contain 
different levels of content scaling for different destination computers. Computers 
with limited resources may not be able to render complex multimedia content com- 
binations. Computers with a slow network connection may not be able to down- 
load/stream large amounts of multimedia data, such as video and audio. With con- 
tent scaling, it is possible to maintain multiple representations of the same content 
for different destination computers. A further weakness of these formats is that 
they do not provide the compactness that is necessary for rapid transmission over 
transport mediums. Such formats do not provide streaming capabilities, so that the 
destination Tenderers can render the multimedia content while the multimedia con- 
tent is being transmitted over the transport medium. 

SUMMARY OF THE INVENTION 

A format is defined and adopted for a logical structure that encapsulates 
and/or describes multimedia content that may include program instruction code for 
controlling the playback of the multimedia content. The multimedia content may be 
of different media. The data of the multimedia content is partitioned into blocks that 
are suitable for transmission over a transport medium. The blocks may include de- 
scription of the multimedia content and/or the multimedia content data. The blocks 
may also include program code that may be interpreted and/or executed on the 
destination Tenderers. The blocks may be compressed and/or encrypted. 

The invention includes a computer system that has a logical structure for 
encapsulating multimedia content that are partitioned into blocks for holding and/or 
describing the multimedia content that may include program instruction code for 
controlling the playback of the multimedia content. A computerized method for cre- 
ating, transmitting, and rendering the content, based on the logical structure, is 
also included. 

In accordance with a first aspect the invention provides; a method, in a 
computer system, of encapsulating multimedia content data, multimedia content 
description data, and program instruction code into an aggregated data represen- 
tation comprising a logical structure, the method comprising storing on a storage 



device, information about the multimedia content data, the multimedia content de- 
scription data, and the program instruction code to form a main header section in 
the logical structure; storing on the storage device, multiple block headers for all 
multimedia content data, multimedia content description data, and the program in- 
5 struction code to form a block headers section in the logical structure; and storing 
on the storage device, multiple data blocks for all multimedia content data, multi- 
media content description data, and the program instruction code to form a data 
blocks section in the logical structure. 

In a preferred embodiment the method further comprising determining the 
10 storing order of the resources, for the different multimedia types, e.g. audio, video, 
image and text, providing efficient streaming transmission; compressing the data 
in some of the data blocks section using appropriate compression schemes, e.g. 
as ZLIB, PNG or JPEG; and providing different scaled content representations of 
one or more scenes, depending on different hardware profiles of the destination 
15 computers, e.g. bitrate, screen, language, and/or machine. 

In a further embodiment the aggregate^ data representation or the logical 
structure are transferred across a transport medium to one or more destination 
computers. Linking between multiple files with multimedia content may be accom- 
plished by using an externaljink field in the block headers section. 
20 According to a second aspect, the invention provides, in a computer sys- 

tem, a method of retrieving multimedia content data, multimedia content descrip- 
tion data, and program instruction code from an aggregated data representation 
stored on a storage device, the data representation comprising a logical structure 
encapsulating the multimedia content data, multimedia content description data, 
25 and program instruction code. The method comprising reading from the storage 
device a main header section of the logical structure, the main header section hav- 
ing information about the multimedia content data, the multimedia content descrip- 
tion data, and the program instruction code; multiple header blocks from the hea- 
der section of the logical structure, the multiple block headers comprising informa- 
30 tion about multimedia content data, multimedia content description data, and pro- 
gram instruction code; and multiple data blocks from the data section in the logical 
structure, the multiple data blocks comprising multimedia content data, multimedia 
content description data, and program instruction code. 
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The method may further comprise receiving the aggregated data represen- 
tation or the logical structure across a transport medium on a destination compu- 
ter, for immediately, or at a later time, rendering the content using a Tenderer. 

In an embodiment the block headers sections comprising a scene block 
header; the block headers sections comprising an image resource block header, a 
text resource block header, a mesh resource block header, or a video resource 
block header; the data blocks section comprising a scene data block; the data 
blocks section comprising an image resource data block, a text resource data 
block, a mesh resource data block, or a video resource data block; the number of 
data blocks In the data blocks section is equal to the number of block headers in 
the block headers section with an empty externaljink field; and the program in- 
struction code controls playback of the multimedia content. The logical structure 
may be a XML formatted structure. 

In a third aspect the invention provides a computer-readable aggregated 
data representation encapsulating multimedia content data, multimedia content 
description data, and program instruction code, the aggregated data representa- 
tion comprising a logical structure stored on a computer readable storage device, 
the logical structure comprising: a main header section comprising information 
about the multimedia content data, multimedia content description data, and pro- 
gram instruction code in a logical structure that defines the aggregated data sepre- 
sentation; a block header section comprising multiple block headers for the multi- 
media content data, multimedia content description data, and program instruction 
code; and a data block section comprising multiple data blocks for all multimedia 
content data, multimedia content description data, and program instruction code. 
The logical structure may also in this case be a XML formatted structure. 

The invention also provides in a further aspect a computer-readable storage 
medium holding instructions for encapsulating multimedia content data, multimedia 
content description data, and program instruction code into an aggregated data re- 
presentation comprising a logical structure, according to the method of encapsulat- 
ing as outlined above. Further in another aspect the invention provides a compu- 
ter-readable storage medium holding instructions for retrieving multimedia content 
data, multimedia content description data, and program instruction code from an 
aggregated data representation stored on a storage device, the data representa- 
tion comprising a logical structure encapsulating the multimedia content data, 
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multimedia content description data, and program instruction code, the instructions 
comprising reading from the storage device: a main header section of the logical 
structure, the main header section having information about the multimedia con- 
tent data, the multimedia content description data, and the program instruction 
code; multiple header blocks from the header section of the logical structure, the 
multiple block headers comprising information about multimedia content data, mul- 
timedia content description data, and program instruction code; and multiple data 
blocks from the data section in the logical structure, the multiple data blocks com- 
prising multimedia content data, multimedia content description data, and program 
instruction code. 

The present invention employs a format (GX) for holding and/or describing 
multimedia content that may include program instruction code for controlling the 
playback of the multimedia content. A GX file/stream may also be referred to as a 
GX movie. A GX movie may contain one or more scenes, and/or one or more re- 
sources, contained in a block-based structure. A scene specifies the content de- 
scription and layout data, and/or the program-instruction code for controlling the 
playback of the multimedia content. A resource may hold specific data items, such 
as images, text, video, etc. FIG. 14 shows an example of two GX files (1400 and 
1401). File 1 contains one image resource ("Image resource 1"), one scene 
("Scene 1"), and links to two resources in File 2. File 2 contains one image re- 
source ("Image resource 2") and ('Text resource 2"). 

GX is well suited for efficient use on any class of computer, from computers 
with Very limited hardware resources (e.g. handheld devices like mobile phones, 
PDA's and set-top boxes for Interactive TV), to computers with powerful hardware 
resources. GX uses a block-based format for holding and/or describing multimedia 
content. Since the block-based format is relatively flat and un-complex, in its data 
structure organization, is easy to process and render on the destination computer. 
This results in a very small Tenderer implementation, and very low use of hardware 
resources, on the destination computer. 

GX is flexible with respect to the different media types and/or program code 
types that it may contain. The block-based structure of the format makes it easy to 
extend with a vast variety of media types. Depending on the value of the type 
field, the header and data blocks may contain a large number of different media 
types, limited only by the different Tenderer implementations. GX provides good 
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support for content scaling. The author can scale the scene with respect to bitrate 
(bandwidth), language (Norwegian, English, etc.), screen (resolution, refresh rate, 
etc.), and machine (computer class). Furthermore the author may split the scaled 
content into multiple files that are linked together using an extemaljink field, 
which is important for rapid loading of a specific content scaling by the destination 
Tenderer. See example in FIG. 14. The figure illustrates two GX files that are linked 
together using the extemaljink. 

GX is very efficient with respect to compactness in holding multimedia con- 
tent. The individual blocks, or data in the blocks, may use different compression 
schemes, such as ZLIB, PNG, or JPEG compression. The author may specify 
which compression scheme to use in the content creation process. 
GX provides streaming transmission, so that the destination Tenderers can render 
the multimedia content while the multimedia content is being transmitted over the 
transport medium. GX uses resources to store the different media types, which the 
scenes use. See examples in FIG. 5, 6, 12, and 13. The figures illustrate how one 
can contain the multimedia content types; image, text, mesh, and video, as resour- 
ces, using block headers and data blocks. The resources may be stored in any 
order by the content creation process, which gives the content author the ability to 
specify in which order the resources should be loaded, when streamed over a 
transmission medium. This is very important on slow transport mediums. See ex- 
ample in FIG. 14. The figure illustrates two GX files that are linked together using 
the extemaljink. The two example files contain resources that are ordered, and 
linked together, for efficient streaming transmission. 

BRIEF DESCRIPTION OF DRAWINGS 

Embodiments of the present invention will now be described with reference 
to the following drawings, where 

FIG. 1 is a block diagram illustrating a computer system that is suitable for practi- 
cing the present invention, 

FIG. 2 is a flowchart illustrating use of the GX format in accordance with an em- 
bodiment of the present invention, 

FIG. 3 is a block diagram illustrating the components of the GX format in accor- 
dance with an embodiment of the present invention, 
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FIG. 4 is a block diagram illustrating the format of the scene_block_header in ac- 
cordance with an embodiment of the present invention, 
FIG. 5 is a block diagram illustrating the format of the 

image_resource_block_header and the text_resource_block_header in ac- 
cordance with an embodiment of the present invention, 
FIG. 6 is a block diagram illustrating the format of the 

mesh_resource_block_header and the video_resource_block_header in ac- 
cordance with an embodiment of the present invention, 
FIG. 7 is a block diagram illustrating the format of the scene_data_block in accor- 
dance with an embodiment of the present invention, 
FIG. 8 is a block diagram illustrating the format of the image_data in accordance 

with an embodiment of the present invention, 
FIG. 9 is a block diagram illustrating the format of the text_data in accordance with 

ah embodiment of the present invention, 
FIG. 10 is a block diagram illustrating the format of the mesh_data in accordance 

with an embodiment of the present invention, 
FIG. 11 is a block diagram illustrating the format of the video_data in accordance 

with an embodiment of the present invention, 
FIG. 12 is a block diagram illustrating the format of the 

image_resource_data_block and the text_resource_data_block in accor- 
dance with an embodiment of the present invention, 
FIG. 13 is a block diagram illustrating the format of the mesh_resource_data_block 
and the video_resource_data_block in accordance with an embodiment of 
the present invention, 
FIG. 14 is a block diagram illustrating an example of two GX files', 
Appendix A is the code listing for the XSD specification of the GXML format, with 

an example GXML formatted file, and 
Appendix B shows the classes used by the program instruction code to control the 
playback. 

DETAILED DESCRIPTION 

FIG. 1 is a block diagram of an illustrative system for practicing an embodi- 
ment of the present invention. The present invention can be practiced on compu- 
ters with very limited hardware resources (CPU, memory, disk, graphics card, 
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etc.), to computers with powerful hardware resources. Computers with limited 
hardware resources may be set-top boxes for Interactive TV and handheld devices 
like cellular phones, PDA's and other devices with CPU and memory, and also 
PC's with CPU, memory, disk, graphics card, and input and output units. FIG. 2 is 
a flowchart that illustrates the steps that are performed in the illustrative embodi- 
ment of FIG. 1. The file content, hereinafter called GX content (104), is built by an 
author (step 200 in FIG. 2) and stored on a storage medium (102) on a source 
computer (100). Sometime later, the GX content (104) is transferred over a tran- 
sport media (105), such as a network connection, to a destination computer (101) 
(step 201 in FIG. 2)> The destination computer (1 01 ) includes a number of Tender- 
ers (103) for rendering the multimedia content that are present within the GX con- 
tent (104). For example, the GX content (104) may include program code, image- 
type data and text-type data. The Tenderers (103) at the destination (101) include 
a program code interpreter, an image Tenderer and a text Tenderer. The interpreter 
and the Tenderers may begin rendering data as soon as they receive data prior to 
the complete transmission of the entire GX content (104) (see step 202 in FIG. 2). 
The interpreters and the Tenderers need not immediately render the data, but rat- 
her may render the data at a later point in time. 

FIG. 3 depicts the basic logical organization of GX content (104). It is up to 
the author to fill in the contents of the GX content in accordance with this format. 
The GX content (104) is divisible into a main header section (300), a block head- 
ers section (301) and a data blocks section (302). In general, the header sections 
(300 and 301) are first transmitted from the source computer (100) to the destina- 
tion computer (101) so that the destination computer may process the information 
within the header section. Subsequently, the data blocks section' (302) is transmit- 
ted from the source computer (100) to the destination computer (101) on a block- 
by-block basis. 

The main header section (300) as illustrated in FIG. 3 contains information 
about the GX content (104). The signature (310) specifies the main type of the GX 
content, and is typically a large number that is unique for a specific authoring en- 
vironment. The byte_count (31 1) specifies the total number of bytes contained in 
the GX content (104). The block_count (312) specifies the total number of blocks 
(external or internal) contained in, or used, by the GX content (104). The 
major_version (313), minor_versibn (314), major_revision (315), and 



minorjrevision (316) specifies the version of the GX content format. The 
extra_data (317) provides extra information about the GX content (104), depend- 
ing on the specific implementation of the GX format. The extra_data (317) is optio- 
nal, and may consist of a variable number of bytes, depending on the specific im- 
plementation. 

Examples of possible data types are indicated in the figures. Here we use 
abbreviations for data types as specified in the C++-programming language, 
"ulong" is short for "unsigned long", "ushort" is short for "unsigned short", "bool" is 
short for "boolean", "string" starts with a unsigned long value indicating the byte 
count of the string followed by the bytes of the UTF-.8 character string, "ulonglong" 
is a 64-bit unsigned long. The invention is not limited to the C++ programming 
language. Other programming languages may also be used. 

The block headers sections (301) as illustrated in FIG. 3 contain a number 
of block headers that provide information about the GX content (1 04). The number 
of block headers is specified by block_count (312) in the main header section 
(300). The information contained in a block header may vary, depending on the 
type of content that it describes. A block header will always begin with the fields as 
indicated in FIG. 3. The type (320) indicates the type of content that the header 
describes; this can for example indicate a scene, an image resource, or a text re- 
source. The byte_count (321 ) specifies the total number of bytes in the block/hea- 
der. The block_byte_count (322) specifies the total number of bytes in the associa- 
ted data block. The name (323) specifies the name of the content item. The 
externaljink (324) specifies a link to the external GX content, in which the associ- 
ated data block is contained. The externaljink is empty if the associated data 
block is contained in the current GX content. The extra_data_1 (325) provides ex- 
tra information about the block header and/or content item, depending on the spe- 
cific implementation of the GX format. The extra_data_1 (325) is optional, and may 
consist of a variable number of bytes, depending on the specific implementation. 
The specific data (326) may contain additional information about the content item. 

The data blocks section (302) as illustrated in FIG. 3 contain a number of 
data blocks that contain the data of the content items in the GX content. The num- 
ber of data blocks in the GX content is equal to the number of block headers in the 
GX content with an empty externaljink- There exists exactly one data block for 
each block header with an empty externaljink in the GX content. The data blocks 
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are specified in the same order, and are of the same content type, as the block 
headers. The type (330) indicates the type of content that the data block contains; 
this can for example indicate a scene, an image resource, or a text resource: The 
byte_count (331) specifies the total number of bytes in the data block. The name 
(332) specifies the name of the content item. The extra_data_1 (333) provides ex- 
tra information about the data block and/or content item, depending on the specific 
implementation of the GX format. The extra_data_1 (333) is optional, and may 
consist of a variable number of bytes, depending on the specific implementation. 
The specific data (334) may contain additional information about the content item. 

The scene content type can be used in GX content to represent the visual 
layout of multiple content items of different types. There can be multiple scenes in 
one GX file. The scene can also be scaled (content scaling) by the Tenderers (103) 
for different representations depending on the characteristics of the destination 
computer (101). The scene_block_header (400) as illustrated in FIG. 4 contains 
the block header data for the associated scene data block. The scene_data_block 
(700) as illustrated in FIG. 7 contains the scene data. The type (320 and 330) indi- 
cates that the type of the content item is of the scene content type. The bitrate_ids 
(41 1 and 71 1) specifies the bitrate identifiers used for content scaling. The 
bitrate_id_count (410 and 710) specifies the number of bitrate identifiers. The 
language_ids (413 and 713) specifies the language identifiers used for content 
scaling. The language_id_count (412 and 712) specifies the number of language 
identifiers. The screenjds (415 and 715) specifies the screen identifiers used for 
content scaling. The screen_id_count (414 and 714) specifies the number of scre- 
en identifiers. The machine_ids (417 and 717) specifies the machine identifiers 
used for content scaling. The machine_id_count (416 and 716) specifies the num- 
ber of machine identifiers. The bitrate_ids, language_ids, screenjds, and. 
machine_ids, may in an embodiment be of the unsigned long data type. The 
extra_data_2 (418 and 718) provides extra information about the scene block 
and/or content item, depending on the specific implementation of the GX format. 
The extra_data_2 (418 and 718) is optional, and may consist of a variable number 
of bytes, depending on the specific implementation. The auto_size (719) specifies 
the layout of the scene inside the scene container. The width (720) and height 
(721 ) specifies the size of the scene. The mouse_pointer (722) specifies how the 
. mouse pointer shall appear on the scene. The back_color (723) specifies the 
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background color of the scene, The back_style (724) specifies the background 
style of the scene. The antialias (725) specifies antialiasing for the scene. The 
quality (726) specifies the quality of the scene rendering. The frames_per_ksec 
(727) specifies the frame-rate of the scene rendering. The program_code (729) 
specifies the program code of the scene. The program_code may begin with an 
unsigned long value indicating the byte count of the program code, and may be 
followed by the bytes of the program. The elemenUcount (731) specifies the byte 
count of the element data. The element_data (732) contains element definitions for 
the scene. The extra_data_3 (728), extra_data_4 (730), and extra_data_5 (733) 
provide extra information about the scene, depending on the specific implementa- 
tion of the GX format. The extra_data_3 (728), extra_data_4 (730), and 
extra_data_5 (733) are optional, and may consist of a variable number of bytes, 
depending on the specific implementation. The program_code (729) can be in any 
programming language or instruction-set, compiled or source code, depending on 
the specific implementation. 

The program code uses the classes; Scene, Image, Text, Mesh, Video, etc., 
as specified in the Java-language in Appendix B. The classes may implement ad- 
ditional functionality, and that there may be more classes, depending on the speci- 
fic implementation. 

The image_data (800) as illustrated in FIG. 8 contain element definition 
data for the scene of the image element type. The text_data (900) as illustrated in 
FIG. 9 contain element definition data for the scene.of the text element type. The 
mesh_data (1000) as illustrated in FIG. 10 contain element definition data for the 
scene of the mesh element type. The video_data (1100) as illustrated in FIG. 11 
contain element definition data for the scene of the video element type. The 
image_data (800), text_data (900), mesh_data (1000), and video_data (1 100) may 
be contained in the element_data (732) of the scene. The left (805, 905, 1005, 
1105), top (806, 906, 1006. 1 106), width (807, 907, 1007, 1107), and height (808, 

908. 1008. 1108) specify the position and size of the element. The rotation (809, 

909. 1009. 1 109) specifies the rotation of the element. The enabled (810, 910, 
101 0, 1 1 10) specifies whether the element is enabled or disabled. The visible 
(81 1 , 91 1 , 1 01 1 , 1 1 1 1 ) specifies whether the element is visible. The transparency 
(812, 912, 1012, 1112) specifies the transparency of the element. The 
mouse_pointer (813, 913, 1013, 1113) specifies how the mouse pointer shall app- 
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ear on the element. The backcolor (814, 914, 1014, 1 1 14) specifies the back- 
ground color of the element. The back_style (81 5, 91 5, 1 01 5, 1 1 15) specifies the 
background style of the element. The extra_data_1 (804, 904, 1004, 1104), and 
extra_data_2 (816, 916, 1016, 1116) provide extra information about the element, 
depending on the specific implementation of the GX format. The extra_data_1 
(804, 904, 1004, 1104), and extra_data_2 (816, 916, 1016, 1116) are optional, and 
may consist of a variable number of bytes, depending on the specific 
implementation. 

The image, text, mesh and/or video resource can be used in GX content to 
contain image, text, 3D mesh and/or video data, respectively. The 
image_resource_block_header (500) as illustrated in FIG. 5 contains the block 
header data for the associated image resource data block. The 
image_resource_data_block (1200) as illustrated in FIG:* 12 contains the image re- 
source data. The text_resource_block_header (550) as illustrated in FIG. 5 con- 
tains the block header data for the associated text resource data block. The 
text_resource_data_block (1250) as illustrated in FIG. 12 contains the text re- 
source data. The mesh_resource_block_header (600) as illustrated in FIG. 6 con- 
tains the block header data for the associated mesh resource data block. The 
mesh_resource_data_block (1300) as illustrated in FIG. .13 contains the mesh re- 
source data. The video_resource_block_header (650) as illustrated in FIG. 6yCon- 
tains the block header data for the associated video resource data block. The 
video_resource_data_block (1350) as illustrated in FIG. 13 contains the video re- 
source data. The imagejype (510 and 1210) specifies the type of the image data. 
The width (51 1 and 1 21 1 ) and height (51 2 and 1 21 2) specifies the size of the 
image. The bit_count (513 and 1213) specifies the number of bits per pixel of the 
image. The resource_data (1215, 1261, 1311, 1361) specifies the data of the re- 
source. The resource_data may begin with an unsigned long value indicating the 
byte count of the resource data, and may be followed by the bytes of the resource 
data. The extra_data_2 (514, 560, 610, 660, 1214, 1260, 1310, 1360), and 
extra_data_3 (1216, 1262, 1312, 1362) provide extra information about the re- 
source, depending on the specific implementation of the GX format. The 
extra_data_2 (514, 560, 610, 660, 1214, 1260, 1310, 1360), and extra_data_3 
(1216, 1262, 1312, 1362) are optional, and may consist of a variable number of 
bytes, depending on the specific implementation. 
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The World Wide Web Consortium (W3C) has defined the Extensible 
Markup Language (XML) universal format for structured documents and data on 
the Web. It is easy to see that the GX format can easily be represented using 
XML. Appendix A shows a XML Schema (XSD), for representing the GX format, 
according to the W3C XSD specifications. At the end of Appendix A is an example 
XML document, containing GX formatted content in XML format, based on the 
XML Schema. The XSD specification in Appendix A specifies the preferred XML 
representation of GX formatted content (GXML). 

FIG. 14 is an example on how content can be effectively linked together for 
the purpose of efficient transmission of the multimedia content over a slow trans- 
mission medium. Typically, the main problems with a slow transmission medium 
are; high access time and low transmission rate. The access time is the time from 
the destination computer requests content, until the destination computer initially 
receives it. The transmission rate is the rate at which data can be delivered across 
the transmission medium. The GX format can embed many small content items as 
resources, which reduces the total content transfer time on transmission mediums 
with a high access time. As one can see in FIG. 14, the GX files (1400 and 1401) 
contain multiple data blocks, which contain content items, in each GXfile. The ar- 
rows in FIG. 14 illustrates content linking, using the extemaljink (324) field of the 
block headers. The extemaljink field indicates where the data block is locajed, 
either in the same file, or an external file. The extemaljink field may be an URL. 
By linking multimedia content in this manner, one can have efficient reuse of multi- 
media content between different GX files, while maintaining a minimal number of 
GX content files. Reuse of multimedia content is important, since it can be used to 
avoid having to retransmit the same content item multiple times'/You do want to 
avoid retransmission of content items on slow transmission mediums. 

While the present invention has been described with reference to an embo- 
diment thereof, those skilled in the art will appreciate that various changes in form 
and detail may be made without departing from the intended scope of the inven- 
tion as defined in the appended claims. The particulars described above are inten- 
ded merely to be illustrative and the scope of the invention is defined by the app- 
ended claims. For example, the present invention may be practiced with a multi- 
media content format that differs from the format described above. Alternative mul- 
timedia content formats may include only a subset of the above-described fields or 
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include additional fields that differ from those described above. Moreover, the 
length of the values held within the fields and the organization of the structures de 
scribed above are not intended to limit the scope of the present invention. 



/ 
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APPENDIX A 



<?xml version= M 1 .0" encoding= M UTF-8 tt ?> 
<xs:schema xmlns:xs= ,1 http://w\AW.w3.org/2001/XMLSchema n e!ementFormDefault= n quaiified 
attributeFormDefault= n unquaiified M > 
<xs:element name="gxmr> . 
<xs:annotation> 
<xs:documentation>GXML Document</xs:documentation> 

</xs:annotation> 
<xs:complexType> 
<xs:sequence> 
<xs:element name="head"> 
<xs:complexType> 
<xs:choice maxOccurs= M unbounded"> 
<xs:e!ement name="sceneHeader t, > 
<xs:complexType> 
<xs:attribute name^ame" type= w xs:string" use="required7> 
<xs:attribute name="externaILink M type="xs:string" use= M optiona!7> 
<xs:attribute name^bitratelDs" type="xs:string" use= n optional7> 
<xs:attribute name^languagelDs" type="xs:string" use="optionai7> 
<xs:attribute name= M screenlDs n type= M xs:string" use="optional7> 
<xs:attribute name= n machinelDs n type= w xs:string" use= M optional7> 
</xs:complexType> 
</xs:element> 

<xs:element name-'imageResourceHeader"> 
<xs:complexType> 
<xs:attribute name="name w type^xsistring** use= M required7> 
<xs:attribute name^externalLin^.type^xsistring" use="optional7> 
*xs:attribute name= n imageType n type= w xs:integer" use="optional7> 
<xs:attribute name="width n type="xs:integer" use= M optional7> 
<xs:attribute name="height" type= ,, xs:integer n use="optional7> 
<xs:attribute name="bitCount M type= f, xs:integer" use= n optional7> 
</xs:complexType> 

</xs:element> / 
<xs:element name= n textResourceHeader n > 
<xs:complexType> 
<xs:attribute name= n name n type= n xs:string" use="required7> 
<xs:attribute name="externalLink" type="xs:string" use="optional7> 
</xs:complexType> 
</xs:element> 

<xs:element name^meshResourceHeader^ 
<xs:complexType> . 
<xs:attribute name="name" type= ,, xs:string ,, use= M required7> 
<xs:attribute name= n externalLink" type= n xs:string" use="optional7> 
</xs:comp!exType> 
</xs:element> 

<xs:element name-VideoResourceHeade^ 
<xs:complexType> 
<xs:attribute name="name M type="xs:string n use="required7> 
<xs:attribute name= M externalLink n type= w xs:string M use="optionai7> 
</xs:complexType> 
</xs:element> 
</xs:choice> 
</xs:complexType> 
</xs:element> 

<xs:element name= n movie"> 
<xs:complexType> 
<xs:choice maxOccurs= M unbounded M > 
<xs:element name^scene'^ 
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<xs:complexType> 
<xs:sequence> 
<xs:choice minbccurs= n O" maxOccurs= M unbounded"> 
<xs:element name="image"> 
<xs:complexType> 
<xs:attribute name= n name n type="xs:string n use= n required7> 
<xs:attribute name="resourceName" type="xs:string" use="optional7> 
<xs:attribute name="left" type="xs:integer" use="optional7> 
<xs:attribute name= n top n type^'xsiinteger" use="optiona!7> 
<xs:attribute name= M width n type^'xsnnteger" use="optional7> 
<xs:attribute name= n height" type= w xs:integer M use="optional7> 
<xs:attribute name="rotation" type= M xs float" use="optional7> 
<xs:attribute name="enabled" type="xs:boo!ean" use= M optionaI7> 
<xs:attribute name="visible n type= M xs:boolean" use= M optional7> 
<xs:attribute name="transparency" type^'xsfloat" use= n optional7> 
<xs:attribute name= ,f mousePointer n type="xs:integer H use="optional7> 
<xs:attribute name^backCoior" type="xs:integer" use= w optional7> 
<xs:attribute name= w backStyle" type= n xs:integer n use="optional7> 
</xs:complexType> 
</xs:element> 
<xs:element name= n text"> 
<xs:complexType> 
<xs:attribute name="name n type="xs:string" use="required7> 
<xs:attribute name="resourceName" type= n xs:string n use= n optional7> 
<xs:attribute name="left n type="xs:integer w use="optional7> 
<xs:attribute name= n top" type="xs:integer n use="optional7> 
<xs:attribute name= M width" type= M xs:integer" use= n optional7> 
<xs:attribute name= n height" type= n xs:ipteger" use="optional7> 
<xs:attribute name= M rotation" type= M xs:float" use="optiona!7> 
<xs:attribute name= n enab!ed n type="xs:boolean n use="optional7> 
<xs:attribute name= n visible M type= n xs:boo!ean n use="optional7> 
<xs:attribute name^transparency" type="xsfloat" use="optional7> 
<xs:attribute name="mousePointer" type="xs:integer" use="optional7> 
<xs:attribute name= w backColor" type="xs:integer" use="optional7> 
<xs:attribute name="backStyte" type^xsiinteger" use="optidnal7> . 
</xs:complexType> 
</xs:element> 

<xs:element name= M mesh M > 
<xs:complexType> 
<xs:attribute name="name" type= n xs:string" use="required7> 
<xs:attribute name= n resourceName n type="xs:string n use="optional7> 
<xs:attribute name= n left" type="xs:integer" use= w optional7> 
<xs:attribute name="top" type= n xs:integer" use="optiona!7> 
<xs:attribute name^'width" type= w xs:integer" use="optionai7^ 
<xs:attribute name= n height" type= n xs:integer" use="optional7> 
<xs:attribute name= n rotation" type= n xs:float" use= w optional7> 
<xs:attribute name="enab1ed n type="xs:boolean" use =,, optional7> 
<xs:attribute name="visible" type="xs:boolean" use= n optional7> 
<xs:attribute name="transparency" type^'xsfloat" use="optional7> 
<xs:attribute name="mousePointer n type="xs:integer" use="optiona!7> 
<xs:attribute name= n backColor" type="xs:integer" use="optional7> 
<xs:attribute name="backStyle" type^'xsiinteger" use="optional7> 
</xs:complexType> 
</xs:element> 
<xs:element name="video"> 
<xs:complexType> 
<xs:attribute name="name" type="xs:string" use="required7> 
<xs:attribute name="resourceName" type="xs:string" use="optionai7> 
<xs:attribute name^'left" type^xsiinteger" use="optional7> 
<xs:attribute name= w top" type="xs:integer" use="optional7> 



17 



<xs:attribute name= n width n type^^srinteger" use= n optiona!7> 
<xs:attribute name= n height" type^'xsiinteger" use= n optional7> 
<xs:attribute name^'rotation" type= n xs:float" use="optiona!7> 
<xs:attribute name= n enabled" type="xs:boolean" use="optional7> 
<xs:attribute name="visib!e" type="xs:bootean" use="optional7> 
<xs:attribute name= n transparency" type= n xs:float M use="optiona!7> 
<xs:attribute name^mousePointer" type= n xs:integer n use="optionai7> 
<xs:attribute name= H backColor" type= M xs:integer" use="optional7> 

. <xs:attribute name="backStyie" type= M xs:integer" use="optional7> 

</xs:comp!exType> 
</xs:element> 
</xs:choice> 

<xs:element name= n program"> 
<xs:complexType> 
<xs:choice minOccurs="0 w maxOccurs= n unbounded"> 
<xs:e!ement name= n hexBinaryData" type="xs:hexBinary7> . 
</xs:choice> 

<xs:attribute name="src" type= w xs:string n use="optional7> 
</xs:comp!exType> 
</xs:element> 
</xs:sequence> 

<xs:attribute name="name w type="xs:string n use="required7> 
<xs:attribute name^bitratelDs" type= n xs:string" use= w optional7> 
<xs:attribute name- w languagelDs" type="xs:string" use="optiona!7> 
<xs:attribute name="screenlDs n type="xs:string" use="optional7> 
<xs:attribute name= n machinelDs n type= n xs:string" use= n optional7> 
<xs:attribute name= n autoSize" type= w xs:integer" use="optional7> 
<xs:attribute name^'width" type= w xs:integer use="optional7> 
<xs:attribute name^eight" type= n xs:integer n use="optional7> 
<xs:attribute name= n mousePointer" type^'xsiinteger" use="optional7> 
<xs:attribute name="backColor" type= w xs:string M use= w optiona!7> 
<xs:attribute name= tt backStyle n type="xs:integer n use= w optionai7> 
<xs:attribute name^antialias" type="xs -.integer" use= M optional7> 
<xs:attribute name= n quality n type^xsiinteger" use= n optional7>. 
<xs:attribute name^ramesPerKSec" type= n xs:integer" use="optional7> 
</xs:complexType> 
</xs:element> 

<xs:element name="imageResource M > 
<xs:complexType> 
<xs:choice minOccurs="0" maxOccurs^unbounded^ 

<xs:element name="hexBinaryData n type= tt xs:hexBinary7> 
</xs:choice> 

<xs:attribute name="name n type= w xs:string" use="required7> 
<xs:attribute name= w imageType" type= M xs:integer" use="optional7> 
<xs:attribute name="width n type="xs:integer w use= n optional7> 
<xs:attribute name= w height" type= n xs:integer" use="optionaI7> 
<xs:attribute name= n bitCount n type^xsrinteger" use= n optional7> 
<xs:attribute name="src n type= w xs:string" use= n optional7> 
</xs:complexType> 
</xs:element> 

<xs:element name^textResource'^ 
<xs:comp!exType> 
<xs:choice minOccurs="0" maxOccurs="unbounded"> 

<xs:element name="hexBinaryData n type="xs:hexBinary7> 
</xs:choice> 

<xs:attribute name="name" type= w xs:string" use="required7> 
<xs:attribute name="src n type= n xs:string" use="optional7> 
</xs:complexType> 
</xs:element> 

<xs:element name= n meshResaurce"> 
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<xs:complexType> 
<xs:choice minOccurs=="O n maxOccurs="unbounded"> 

<xs:e!ement name="hexBinaryData" type="xs:hexBinary7> 
</xs:choice> 

<xs:attribute name="name" type="xs:string" use= ,, required , V> 
<xs:attribute name^src" type="xs:string" use="optiona!7> 
</xs:complexType> 
</xs:element> 

<xs:element name= ,, videoResource"> 
<xs:comp!exType> 
<xs:choice minOccurs="0" maxOccurs= ,, unbounded"> 
<xs:element name= n hexBinaryData" type="xs:hexBinary'V> 
</xs:choice> 

<xs:attribute name^name" type= n xs:string" use= n required7> 
<xs:attribute name= n src" type= n xs:string" use="optional7> 
</xs:complexType> 
</xs:element> 
</xs:choice> 
</xs:complexType> 
</xs:element> 
</xs:sequence> 
</xs:comp!exType> 
</xs:element> 
</xs:schema> 



<?xml verston="1 .0" encoding^UTF-S"?^ 

<gxml xmlns:xsi= M http://www.w3.org/2001 /XMLSchema-instance"> 
<head> 

<sceneHeader name="Scener externalLink= ,m bitratelDs="56000 n languagelDs= n 0" 
screenlDs= n 07> _ M >JU _ ncn 

<imageResourceHeader name="IRJmage1" externalLink^ imageType= 0 width= 5 

height="5" bitCount="247> 
</head> 

<movie>. ____ 
<scene name.= n Scener bitratelDs="56000" languagelDs="O n screenlDs= 0 
autoSize="0" width="550 n height="400" mousePointer= n 0" backColor^O" backStyle= 0 
antialias= n 0" quality= w 0 n framesPerKSec= w 250(r> 

<image name="lmage1" resourceName= B IRJmage1 M left="20" top= M 20" width= w 5 rt 

height="5" rotation= tt 0" enabled="true n visible="true" transparency="O n 

mousePointer= n O n backColor="0" backSty!e="07> 

<program src=" n > 

<hexBinaryData>O00O0O0OOO0O0O00O0O00OO0OfffOffT0ffTO00O0O0OOfff0q0OOfff 
OOOOOOOOOfff0fffOfff0OOOO0OO0O0OO0O00000OOOO</hexBinaryData> 

</program> 
</sc©na^ 

<imageResource name= n IRJmage1 M imageType="0" width= n 5" height= n 5" 

bitCount="24 w src=""> ^ 
<hexBinaryData>0O00OO000OO0O0O00000O0000fff0fffOflT00O0O0O0Offf0000Om 

OO0OOOOOOfffOfffOfffO0OO000OO0O0O00O0O0O0O0O</hexBinaryData> 

</imageResource> 
</movie> 
</gxml> 
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APPENDIX B 



public class Scene 

5 public final static native int getWidth(); 
public final static native int getHeight(); 
public final static native void p!ayScene(String name); 

} 



public class Image 
{ 

public int getLeft(); 

public void setLeft(int value); 
15 public int getTopQ; 

public void setTop(int value); 

public int getWidth(); 

public void setWidth(int value); 

public int getHeight(); 
20 public void setHeight(int value); 

public float getRotation(); 

public void setRotation(float value); 

public boolean getEnabted(); 

public void setEnabled(boolean value); 
25 public boolean getVisible(); 

public void setVisible(boolean value); 

public float getTransparency(); 

public void setTransparency(float value); 

} 

30 • 

public class Text 
{ 

public int getLeft(); 
35 public void setLeft(int value); 

public int getTop(); 

public void setTop(int value); 

public int getWidth(); 

public void setWidth(int value); 
40 public int getHeight(); 

public void setHeight(int value); 

public float getRotation(); 

public void setRotation(float value); 

public boolean getEnabled(); 
45 public void setEnabled(boolean value); 

public boolean getVisibleQ; 

public void setVisible(boolean value); 

public float getTransparency(); 

public void setTransparency(float value); 

so } 



public class Mesh 
{ 

55 public int getLeft(); 

public void setLeft(int value); 

public int getTop(); 

public void setTop(int value); 
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public int getWidth(); 

public void setWidth(int value); 

public int getHeightQ; 

public void setHeight(int value); 

public float getRptation(); 

public void setRotation(float value); 

public boolean getEnabled(); 

public void setEnabled(boolean value); 

public boolean getVisible(); 

public void setVisible(boolean value); 

public float getTransparency(); 

public void setTransparency(float value); 

} 



public class Video 
{ 

public int getLeft(); 

public void setLeft(int value); 

public int getTop(); 

public void setTop(int value); 

public int getWidth(); . 

public void setWidth(int value); 

public int getHeight(); 

public void setHeight(int value); 

public float getRotation(); 

public void setRotation(float value); 

public boolean getEnabled(); 

public void setEnabled(boolean value); 

public boolean getVisible(); 

public void setVisible(boolean value); 

public float getTransparency(); 

public void setTransparency(float value); 

} 
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CLAIMS 

1 . In a computer system, a method of encapsulating multimedia content data, 
multimedia content description data, and program instruction code into an aggre- 
5 gated data representation comprising a logical structure, the method comprising: 

- storing on a storage device, information about the multimedia content data, the 
multimedia content description data, and the program instruction code to form a 
main header section (300) in the logical structure; 

- storing on the storage device, multiple block headers for all multimedia content 
io data, multimedia content description data, and the program instruction code to 

form a block headers section (301) in the logical structure; and 

- storing on the storage device, multiple data blocks for all multimedia content 
data, multimedia content description data, and the program instruction code to 
form a data blocks section (302) in the logical structure. 

15 

2. Method according to claim 1, wherein: 

- the block headers sections (301 ) comprising a scene block header (400); 

- the block headers sections (301) comprising an image resource block header 
(500), a text resource block header (550), a mesh resource block header (600), or 

20 a video resource block header (650); / 

- the data blocks section (302) comprising a scene data block (700); 

-the data blocks section (302) comprising an image resource data block (1200), a 
text resource data block (1250), a mesh resource data block (1300), or a video re- 
source data block (1350); 
25 - the number of data blocks in the data blocks section (302) is equal to the number 
of block headers in the block headers section (301) with an empty externaljink 
field (324); and 

- the program instruction code controls playback of the multimedia content. 

30 3. Method according to claim 1 , further comprising: 

- determining the storing order of the resources, for the different multimedia types, 
e.g. audio, video, image and text, providing efficient streaming transmission; 

- compressing the data in some of the data blocks section using appropriate com- 
pression schemes, e.g. as ZLIB, PNG or JPEG; and 




- providing different scaled content representations of one or more scenes, depen- 
ding on different hardware profiles of the destination computers (101), e.g. bitrate, 
screen, language, and/or machine. 

4. Method according to claim 1 , wherein the logical structure is a XML format- 
ted structure. 

5. Method according to claim 1 , further comprising transferring the aggregated 
data representation or the logical structure across a transport medium (105) to one 
or more destination computers (101). 

6. Method according to claim 3, further comprising providing linking between 
multiple files with multimedia content by use of an exterhaljink field (324) in the 
block headers section (301 ). 

7. In a computer system, a method of retrieving multimedia content data, mul- 
timedia content description data, and program instruction code from an aggrega- 
ted data representation stored on a storage device, the data representation com- 
prising a logical structure encapsulating the multimedia content data, multimedia 
content description data, and program instruction code, the method comprising 
reading from the storage device: 

- a main header section (300) of the logical structure, the main header section hav- 
ing information about the multimedia content data, the multimedia content descrip- 
tion data, and the program instruction code; 

- multiple header blocks from the header section (301) of the logfcal structure, the 
multiple block headers comprising information about multimedia content data, mul- 
timedia content description data, and program instruction code; and 

- multiple data blocks from the data section (302) in the logical structure, the multi- 
ple data blocks comprising multimedia content data, multimedia content descrip- 
tion data, and program instruction code. 
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8. Method according to claim 7, wherein: 

- the block headers sections (301) comprising a scene block header (400); 

- the block headers sections (301) comprising an image resource block header 
(500), a text resource block header (550), a mesh resource block header (600), or 
a video resource block header (650); 

- the data blocks section (302) comprising a scene data block (700); 

- the data blocks section (302) comprising an image resource data block (1200), a 
text resource data block (1250), a mesh resource data block (1300), or a video re- 
source data block (1350); 

- the number of data blocks in the data blocks section (302) is equal to the number 
of block headers in the block headers section (301 ) with an empty externaljink 
field (324); and 

- the program instruction code controls playback of the multimedia content. 

9. Method according to claim 7, wherein the logical structure is a XML format- 
ted structure. 

1 0. Method according to claim 7, further comprising receiving the aggregated 
data representation or the logical structure across a transport medium (105) on a 
destination computer (101), for immediately, or at a later time, rendering the,con- 
tent using a rehderer (1 03). 

11. Computer-readable aggregated data representation encapsulating multime- 
dia content data, multimedia content description data, and program instruction 
code, the aggregated data representation comprising a logical structure stored on 
a computer readable storage device, the logical structure comprising: 

- a main header section (300) comprising information about the multimedia content 
data, multimedia content description data, and program instruction code in a logi- 
cal structure that defines the aggregated data representation; 

- a block header section (301 ) comprising multiple block headers for the multime- 
dia content data, multimedia content description data, and program instruction 
code; and 

- a data block section (302) comprising multiple data blocks for all multimedia con- 
tent data, multimedia content description data, and program instruction code. 
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1 2. Computer-readable aggregated data representation of claim 1 1 , wherein: 

- the block headers sections (301 ) comprising a scene block header (400); 

- the block headers sections (301) comprising an image resource block header 

5 (50.0), a text resource block header (550), a mesh resource block header (600), or 
a video resource block header (650); 

- the data blocks section (302) comprising a scene data block (700); 

- the data blocks section (302) comprising an image resource data block (1200), a 
text resource data block (1250), a mesh resource data block (1300), or a video re- 

io source data block (1350); 

- the number of data blocks in the dafa blocks section (302) is equal to the number 
of block headers in the block headers section (301) with an empty externaljink 
field (324); and 

- the program instruction code controls playback of the multimedia content. 

15 

1 3. Computer-readable aggregated data representation of claim 1 1 , wherein 
the logical structure is a XML formatted structure. 

14. A computer-readable storage medium holding instructions for encapsulating 
20 multimedia content data, multimedia content description data, and program in- 
struction code into an aggregated data.representation comprising a logical struc- 
ture, the instructions comprising: 

- storing on a storage device, information about the multimedia content data, the 
multimedia content description data, and the program instruction code to form a 

25 main header section (300) in the logical structure; 

- storing on the storage device, multiple block headers for all multimedia content 
data, multimedia content description data, and the program instruction code to 
form a block headers section (301 ) in the logical structure; and 

- storing on the storage device, multiple data blocks for all multimedia content 
30 data, multimedia content description data, and the program instruction code to 

form a data blocks section (302) in the logical structure. 
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1 5. A computer-readable storage medium holding instructions for retrieving 
multimedia content data, multimedia content description data, and program in- 
struction code.from an aggregated data representation stored on a storage device, 
the data representation comprising a logical structure encapsulating the multime- 
dia content data, multimedia content description data, and program instruction 
code, the instructions comprising reading from the storage device: 

- a main header section (300) of the logical structure, the main header section hav- 
ing information about the multimedia content data, the multimedia content descrip- 
tion data, and the program instruction code; 

- multiple. header blocks from the header section (301) of the logical structure, the 
multiple block headers comprising information about multimedia content data, mul- 
timedia content description data, and program instruction code; and 

- multiple data blocks from the data section (302) in the logical structure, the multi- 
ple data blocks comprising multimedia content data, multimedia content descrip- 
tion data, and program instruction code. 
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ABSTRACT 

A computerized method of encapsulating multimedia content data, multime- 
dia content description data, and program instruction code into an aggregated data 
representation comprising a logical structure is described. The method comprising: 

- storing on a storage device, information about the multimedia content data, the 
multimedia content description data, and the program instruction code to form a 
main header section (300) in the logical structure; 

- storing on the storage device, multiple block headers for all multimedia content 
data, multimedia content description data, and the program instruction code to 
form a block headers section (301) in the logical structure; and 

- storing on the storage device, multiple data blocks for all multimedia content 
data, multimedia content description data, and the program instruction code to 
form a data blocks section (302) in the logical structure. . 
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